{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center/> 自定义调试体验文档"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本文将使用[快速入门](https://gitee.com/mindspore/docs/blob/master/tutorials/tutorial_code/lenet/lenet.py)作为样例，并通过构建自定义调试函数：`Callback`、`metrics`、Print算子、日志打印、数据Dump功能等，同时将构建的自定义调试函数添加进代码中，通过运行效果来展示具体如何使用MindSpore提供给我们的自定义调试能力，帮助快速调试训练网络。\n",
    "体验过程如下：\n",
    "1. 数据准备。\n",
    "2. 定义深度神经网络LeNet5。\n",
    "3. 使用Callback回调函数构建StopAtTime类来控制训练停止时间。\n",
    "4. 设置日志环境变量。\n",
    "5. 启动同步Dump功能。\n",
    "5. 定义训练网络并执行训练。\n",
    "6. 执行测试。\n",
    "7. 算子输出数据的读取与展示。\n",
    "\n",
    "> 本次体验适用于GPU环境。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据准备"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集的下载"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们需要将MNIST数据集中随机取出一张图片，并增强成适合LeNet网络的数据格式（如何处理请参考[mindspore_quick_start.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/training/source_zh_cn/quick_start/quick_start.ipynb)）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行以下命令来获取数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./datasets/MNIST_Data\n",
      "├── test\n",
      "│   ├── t10k-images-idx3-ubyte\n",
      "│   └── t10k-labels-idx1-ubyte\n",
      "└── train\n",
      "    ├── train-images-idx3-ubyte\n",
      "    └── train-labels-idx1-ubyte\n",
      "\n",
      "2 directories, 4 files\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p ./datasets/MNIST_Data/train ./datasets/MNIST_Data/test\n",
    "!wget -NP ./datasets/MNIST_Data/train https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-labels-idx1-ubyte \n",
    "!wget -NP ./datasets/MNIST_Data/train https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-images-idx3-ubyte\n",
    "!wget -NP ./datasets/MNIST_Data/test https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-labels-idx1-ubyte\n",
    "!wget -NP ./datasets/MNIST_Data/test https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-images-idx3-ubyte\n",
    "!tree ./datasets/MNIST_Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`custom_debugging_info.ipynb`为本文文档。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集的增强操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下载的数据集，需要通过`mindspore.dataset`处理成适用于MindSpore框架的数据，再使用一系列框架中提供的工具进行数据增强操作来适应LeNet网络的数据处理需求。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore.dataset as ds\n",
    "import mindspore.dataset.vision.c_transforms as CV\n",
    "import mindspore.dataset.transforms.c_transforms as C\n",
    "from mindspore.dataset.vision import Inter\n",
    "from mindspore import dtype as mstype\n",
    "\n",
    "def create_dataset(data_path, batch_size=32, repeat_size=1,\n",
    "                   num_parallel_workers=1):\n",
    "    \"\"\" create dataset for train or test\n",
    "    Args:\n",
    "        data_path (str): Data path\n",
    "        batch_size (int): The number of data records in each group\n",
    "        repeat_size (int): The number of replicated data records\n",
    "        num_parallel_workers (int): The number of parallel workers\n",
    "    \"\"\"\n",
    "    # define dataset\n",
    "    mnist_ds = ds.MnistDataset(data_path)\n",
    "\n",
    "    # define operation parameters\n",
    "    resize_height, resize_width = 32, 32\n",
    "    rescale = 1.0 / 255.0\n",
    "    shift = 0.0\n",
    "    rescale_nml = 1 / 0.3081\n",
    "    shift_nml = -1 * 0.1307 / 0.3081\n",
    "\n",
    "    # define map operations\n",
    "    trans_image_op=[\n",
    "        CV.Resize((resize_height, resize_width), interpolation=Inter.LINEAR),\n",
    "        CV.Rescale(rescale_nml, shift_nml),\n",
    "        CV.Rescale(rescale, shift),\n",
    "        CV.HWC2CHW() \n",
    "    ]\n",
    "    type_cast_op = C.TypeCast(mstype.int32)\n",
    "\n",
    "    # apply map operations on images\n",
    "    mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns=\"label\", num_parallel_workers=num_parallel_workers)\n",
    "    mnist_ds = mnist_ds.map(operations=trans_image_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n",
    "\n",
    "    # apply DatasetOps\n",
    "    buffer_size = 10000\n",
    "    mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size)\n",
    "    mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)\n",
    "    mnist_ds = mnist_ds.repeat(repeat_size)\n",
    "\n",
    "    return mnist_ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义深度神经网络LeNet5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "针对MNIST数据集我们采用的是LeNet5网络，先对卷积函数和全连接函数初始化，然后`construct`构建神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindspore.common.initializer import Normal\n",
    "import mindspore.nn as nn\n",
    "\n",
    "class LeNet5(nn.Cell):\n",
    "    \"\"\"Lenet network structure.\"\"\"\n",
    "    def __init__(self):\n",
    "        super(LeNet5, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5, pad_mode=\"valid\")\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5, pad_mode=\"valid\")\n",
    "        self.fc1 = nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02))\n",
    "        self.fc2 = nn.Dense(120, 84, weight_init=Normal(0.02))\n",
    "        self.fc3 = nn.Dense(84, 10)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "        self.flatten = nn.Flatten()\n",
    "        \n",
    "    def construct(self, x):\n",
    "        x = self.max_pool2d(self.relu(self.conv1(x)))\n",
    "        x = self.max_pool2d(self.relu(self.conv2(x)))\n",
    "        x = self.flatten(x)\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建自定义回调函数StopAtTime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用回调函数的基类Callback，构建训练定时器`StopAtTime`，其基类（可在源码中找到位置在`/mindspore/nn/callback`）为："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "class Callback():\n",
    "    def begin(self, run_context):\n",
    "        pass\n",
    "    def epoch_begin(self, run_context):\n",
    "        pass\n",
    "    def epoch_end(self, run_context):\n",
    "        pass\n",
    "    def step_begin(self, run_context): \n",
    "        pass\n",
    "    def step_end(self, run_context):\n",
    "        pass\n",
    "    def end(self, run_context):\n",
    "        pass\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `begin`：表示训练开始时执行。\n",
    "- `epoch_begin`：表示每个epoch开始时执行。\n",
    "- `epoch_end`：表示每个epoch结束时执行。\n",
    "- `step_begin`：表示每个step刚开始时执行。\n",
    "- `step_end`：表示每个step结束时执行。\n",
    "- `end`：表示训练结束时执行。\n",
    "\n",
    "了解上述基类的用法后，还有一个参数`run_context`，这是一个类，存储了模型训练中的各种参数，我们在这里使用`print(cb_params.list_callback)`将其放在`end`中打印（当然也可以使用`print(cb_param)`打印所有参数信息，由于参数信息太多，我们这里只选了一个参数举例），后续在执行完训练后，根据打印信息，会简单介绍`run_context`类中各参数的意义，我们开始构建训练定时器，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindspore.train.callback import Callback\n",
    "import time\n",
    "\n",
    "class StopAtTime(Callback):\n",
    "    def __init__(self, run_time):\n",
    "        super(StopAtTime, self).__init__()\n",
    "        self.run_time = run_time*60\n",
    "\n",
    "    def begin(self, run_context):\n",
    "        cb_params = run_context.original_args()\n",
    "        cb_params.init_time = time.time()\n",
    "    \n",
    "    def step_end(self, run_context):\n",
    "        cb_params = run_context.original_args()\n",
    "        epoch_num = cb_params.cur_epoch_num\n",
    "        step_num = cb_params.cur_step_num\n",
    "        loss = cb_params.net_outputs\n",
    "        cur_time = time.time()\n",
    "        if (cur_time - cb_params.init_time) > self.run_time:\n",
    "                print(\"epoch: \", epoch_num, \" step: \", step_num, \" loss: \", loss)\n",
    "                run_context.request_stop()\n",
    "    def end(self, run_context):\n",
    "        cb_params = run_context.original_args()\n",
    "        print(cb_params.list_callback)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 启动同步Dump功能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本例中使用同步Dump功能，导出每次迭代中前向传播和反向传播算子的输出数据，导出的数据方便用户在进行优化训练策略时进行分析使用，如需导出更多数据可参考[官方教程](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_debugging_info.html#dump)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "\n",
    "abspath = os.getcwd()\n",
    "\n",
    "data_dump = {\n",
    "        \"common_dump_settings\": {\n",
    "        \"dump_mode\": 0,\n",
    "        \"path\": abspath + \"/data_dump\",\n",
    "        \"net_name\": \"LeNet5\",\n",
    "        \"iteration\": 0,\n",
    "        \"input_output\": 2,\n",
    "        \"kernels\": [\"Default/network-WithLossCell/_backbone-LeNet5/flatten-Flatten/Reshape-op118\"],\n",
    "        \"support_device\": [0,1,2,3,4,5,6,7]\n",
    "    },\n",
    "        \"e2e_dump_settings\": {\n",
    "        \"enable\": True,\n",
    "        \"trans_flag\": False\n",
    "    }\n",
    "}\n",
    "\n",
    "with open(\"./data_dump.json\", \"w\", encoding=\"GBK\") as f:\n",
    "    json.dump(data_dump, f)\n",
    "    \n",
    "os.environ['MINDSPORE_DUMP_CONFIG'] = abspath + \"/data_dump.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "执行完上述命令后会在工作目录上生成`data_dump.json`文件，目录结构如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "├── data_dump.json\n",
      "├── datasets\n",
      "│   └── MNIST_Data\n",
      "│       ├── test\n",
      "│       │   ├── t10k-images-idx3-ubyte\n",
      "│       │   └── t10k-labels-idx1-ubyte\n",
      "│       └── train\n",
      "│           ├── train-images-idx3-ubyte\n",
      "│           └── train-labels-idx1-ubyte\n",
      "├── mindspore_custom_debugging_info.ipynb\n",
      "└── MNIST_Data.zip\n",
      "\n",
      "4 directories, 7 files\n"
     ]
    }
   ],
   "source": [
    "! tree ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "启动同步Dump功能需要注意：\n",
    "\n",
    "- `path`需要设置成绝对路径。例如`/usr/data_dump`可以，`./data_dump`则不行。\n",
    "- `e2e_dump_settings`中的`enable`需要设置成`True`。\n",
    "\n",
    "- 需要将生成的`data_dump.json`文件添加至系统环境变量中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 设置日志环境变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MindSpore采用`glog`来输出日志，我们这里将日志输出到屏幕：\n",
    "\n",
    "`GlOG_v`：控制日志的级别，默认值为2，即WARNING级别，对应关系如下：0-DEBUG、1-INFO、2-WARNING、3-ERROR。本次设置为1。\n",
    "\n",
    "`GLOG_logtostderr`：控制日志输出方式，设置为`1`时，日志输出到屏幕；值设置为`0`时，日志输出到文件。设置输出屏幕时，日志部分的信息会显示成红色，设置成输出到文件时，会在`GLOG_log_dir`路径下生成`mindspore.log`文件。\n",
    "\n",
    "> 更多设置请参考官网：<https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_debugging_info.html>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'GLOG_v': '1', 'GLOG_logtostderr': '1'}\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from mindspore import log as logger\n",
    "\n",
    "os.environ['GLOG_v'] = '1'\n",
    "os.environ['GLOG_logtostderr'] = '1'\n",
    "os.environ['GLOG_log_dir'] = 'D:/' if os.name==\"nt\" else '/var/log/mindspore'\n",
    "os.environ['logger_maxBytes'] = '5242880'\n",
    "os.environ['logger_backupCount'] = '10'\n",
    "print(logger.get_log_config())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印信息为`GLOG_v`的等级：`INFO`级别。\n",
    "\n",
    "输出方式`GLOG_logtostderr`：`1`表示屏幕输出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义训练网络并执行训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义训练网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此过程中先将之前生成的模型文件`.ckpt`和`.meta`的数据删除，并将模型需要用到的参数配置到`Model`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindspore import context, Model\n",
    "from mindspore.nn import Accuracy\n",
    "from mindspore.nn import SoftmaxCrossEntropyWithLogits\n",
    "from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor\n",
    "\n",
    "# clean files\n",
    "if os.name == \"nt\":\n",
    "    os.system('del/f/s/q *.ckpt *.meta')\n",
    "else:\n",
    "    os.system('rm -f *.ckpt *.meta *.pb')\n",
    "\n",
    "context.set_context(mode=context.GRAPH_MODE, device_target=\"GPU\")\n",
    "lr = 0.01\n",
    "momentum = 0.9 \n",
    "epoch_size = 3\n",
    "train_data_path = \"./datasets/MNIST_Data/train\"\n",
    "eval_data_path = \"./datasets/MNIST_Data/test\"\n",
    "model_path = \"./models/ckpt/custom_debugging_info/\"\n",
    "\n",
    "net_loss = SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n",
    "repeat_size = 1\n",
    "network = LeNet5()\n",
    "\n",
    "metrics = {\n",
    "    'accuracy': nn.Accuracy(),\n",
    "    'loss': nn.Loss(),\n",
    "    'precision': nn.Precision(),\n",
    "    'recall': nn.Recall(),\n",
    "    'f1_score': nn.F1()\n",
    "    }\n",
    "net_opt = nn.Momentum(network.trainable_params(), lr, momentum)\n",
    "\n",
    "config_ck = CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10)\n",
    "\n",
    "ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_lenet\", directory=model_path, config=config_ck)\n",
    "\n",
    "model = Model(network, net_loss, net_opt, metrics=metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 执行训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在构建训练网络中，给`model.train`传入了三个回调函数，分别是`ckpoint_cb`，`LossMonitor`，`stop_cb`；其分别代表如下：\n",
    "\n",
    "`ckpoint_cb`：即是`ModelCheckpoint`，设置模型保存的回调函数。\n",
    "\n",
    "`LossMonitor`：loss值监视器，打印训练过程每步的loss值。\n",
    "\n",
    "`stop_cb`：即是`StopAtTime`，上面刚构建的训练定时器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将训练定时器`StopAtTime`设置成36秒，即`run_time=0.6`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============== Starting Training ==============\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] ME(14134:140231287715648,MainProcess):2020-12-01-17:12:19.263.012 [mindspore/train/serialization.py:379] Execute save the graph process.\n",
      "[INFO] ME(14134:140231287715648,MainProcess):2020-12-01-17:12:29.689.876 [mindspore/train/serialization.py:168] Execute save checkpoint process.\n",
      "[INFO] ME(14134:140231287715648,MainProcess):2020-12-01-17:12:29.704.062 [mindspore/train/serialization.py:214] Save checkpoint process finish.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:  1  step:  30  loss:  2.3048654\n",
      "[<mindspore.train.callback._checkpoint.ModelCheckpoint object at 0x7f8a1c116350>, <mindspore.train.callback._loss_monitor.LossMonitor object at 0x7f8997005150>, <__main__.StopAtTime object at 0x7f8a1f042950>]\n"
     ]
    }
   ],
   "source": [
    "print(\"============== Starting Training ==============\")\n",
    "ds_train = create_dataset(train_data_path, repeat_size = repeat_size)\n",
    "stop_cb = StopAtTime(run_time=0.6)\n",
    "model.train(epoch_size, ds_train, callbacks=[ckpoint_cb, LossMonitor(375), stop_cb], dataset_sink_mode=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上打印信息中，主要分为两部分：\n",
    "- 日志信息部分：\n",
    "    - `[INFO]`部分信息即为日志输出的信息，由于没有Warning信息，目前主要记录的是训练的几个重要步骤。\n",
    "    \n",
    "- 回调函数信息部分：\n",
    "    - `LossMonitor`：每步的loss值。\n",
    "    - `StopAtTime`：在每个epoch结束及训练时间结束时，打印当前epoch的训练总时间(单位为毫秒)，每步训练花费的时间以及平均loss值，另外在训练结束时还打印了`run_context.list_callback`的信息，这条信息表示本次训练过程中使用的回调函数；另外`run_conext.original_args`中还包含以下参数：\n",
    "        - `train_network`：网络的各类参数。\n",
    "        - `epoch_num`：训练的epoch数。\n",
    "        - `batch_num`：一个epoch的step数。\n",
    "        - `mode`：MODEL的模式。\n",
    "        - `loss_fn`：使用的损失函数。\n",
    "        - `optimizer`：使用的优化器。\n",
    "        - `parallel_mode`：并行模式。\n",
    "        - `device_number`：训练卡的数量。\n",
    "        - `train_dataset`：训练的数据集。\n",
    "        - `list_callback`：使用的回调函数。\n",
    "        - `train_dataset_element`：打印当前batch的数据集。\n",
    "        - `cur_step_num`：当前训练的step数。\n",
    "        - `cur_epoch_num`：当前的epoch。\n",
    "        - `net_outputs`：网络返回值。\n",
    "\n",
    "   几乎在训练中的所有重要数据，都可以从Callback中取得，所以Callback也是在自定义调试中比较常用的功能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 执行测试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试网络中我们的自定义函数`metrics`将在`model.eval`中被调用，除了模型的预测正确率外`recall`，`F1`等不同的检验标准下的预测正确率也会打印出来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============== Starting Testing ==============\n",
      "============== Accuracy:{'accuracy': 0.10106169871794872, 'loss': 2.302597688558774, 'precision': array([0.       , 0.       , 0.       , 0.1010617, 0.       , 0.       ,\n",
      "       0.       , 0.       , 0.       , 0.       ]), 'recall': array([0., 0., 0., 1., 0., 0., 0., 0., 0., 0.]), 'f1_score': array([0.        , 0.        , 0.        , 0.18357136, 0.        ,\n",
      "       0.        , 0.        , 0.        , 0.        , 0.        ])} ==============\n"
     ]
    }
   ],
   "source": [
    "print(\"============== Starting Testing ==============\")\n",
    "ds_eval = create_dataset(eval_data_path, repeat_size=repeat_size)\n",
    "acc = model.eval(ds_eval,dataset_sink_mode = False)\n",
    "print(\"============== Accuracy:{} ==============\".format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Accuracy`部分的信息即为`metric`控制输出的信息，模型的预测值正确率和其他标准下验证（0-9）的正确率值，至于不同的验证标准计算方法，大家可以去官网搜索`mindspore.nn`查找，这里就不多介绍了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 算子输出数据的读取展示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "执行完成上述训练后，可以在`data_dump`文件夹中找到导出的训练数据，按照本例`data_dump.json`文件的设置，在目录`data_dump/LeNet5/device_0/`中找到每次迭代的数据，保存每次迭代的数据文件夹名称为`iteration_{迭代次数}`，每个算子输出数据的文件后缀为`.bin`，可以使用`numpy.fromfile`读取其中的数据。\n",
    "\n",
    "本例子，在第400次迭代数据中，随机读取其中一个算子的输出文件并进行展示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ops name: fc2.weight_output_0_shape_84_120_Float32_DefaultFormat.bin \n",
      "\n",
      "ops output value: [-1.86227040e-17  7.49122057e-21 -5.01539318e-16 ... -6.28152809e-20\n",
      "  7.43756225e-16  3.97661325e-20] \n",
      "\n",
      "the shape of ops output: (5040,)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "dump_data_path = \"./data_dump/LeNet5/device_0/iteration_400/\"\n",
    "ops_output_file = random.choice(os.listdir(dump_data_path))\n",
    "print(\"ops name:\", ops_output_file, \"\\n\")\n",
    "ops_dir = dump_data_path + ops_output_file\n",
    "ops_output = np.fromfile(ops_dir)\n",
    "print(\"ops output value:\", ops_output, \"\\n\")\n",
    "print(\"the shape of ops output:\", ops_output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "本例使用了MNIST数据集，通过LeNet5神经网络进行训练，将自定义调试函数结合到代码中进行调试，同时展示了使用方法和部分功能，并使用调试函数导出需要的输出数据，来更好的认识自定义调试函数的方便性，以上就是本次的体验内容。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore-1.0.1",
   "language": "python",
   "name": "mindspore-1.0.1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
